Thoughts on keying part 3- Compositing
In part one of this rambling chain of thought, I looked at the way that technology has improved the process of keying blue/greenscreen footage. Compared to 15 years ago we have better cameras, better hardware and better software. In part 2 I looked at keying from a producer’s perspective and concluded that even though technology has advanced, the practical and financial side of video production means that keying will never be as simple as a single click. Good keying will always involve some rotoscoping, and it isn’t realistic to expect to get ‘perfect’ keys with one click of the mouse.
The myth-of-the-single-click is something I ponder a lot. The term itself is something I just made up when I was typing the last part, so perhaps I should elaborate. When I talk about “myth-of-the-single-click” I’m referring to the notion that you can import some blue/greenscreen footage into After Effects, and with one click of the mouse have a perfectly keyed result that’s ready to render. The reason I call it a myth is because this never happens, but the idea that it’s supposed to seems to have perpetuated itself through tutorials, adverts, product demonstrations, and some sort of general industry zeitgeist. As I mentioned in parts 1 & 2, when I started my career I assumed that anyone using a high-end system could key with a single click, and whenever I had to resort to complex workflows in order to get a usable result I assumed it was because something was wrong. And it wasn’t just me, it was my friends and colleagues, and not just those who worked in post – it was also Producers and Directors.
The myth of the single click encapsulates a range of issues that span technology, personality, politics, economics and experience – but it’s not just slick product demonstrations and fancy ads combined with a slight inferiority complex. The myth-of-the-single-click, which I still think confounds corporate video producers and younger people today, also stems from the fact that there’s more to keying than just keying. So let’s look at compositing.
As a job description, keying is really compositing. When I’m hired to do blue/greenscreen keying what I’m really hired to do is composite a scene together. Although my workflow may involve an intermediate stage of rendering a quicktime with an alpha channel, I’ve never been hired to simply key out a background and deliver video with an alpha. Instead, I’m hired to deliver a finished scene. A composite.
It’s fairly simple to judge the quality of a raw key, because it’s pretty clear what the goal is. If you’ve shot someone in front of a greenscreen, then the goal is to remove the green. Judging a composite is more difficult, however, because there’s so many different aspects to a scene to consider. If a composite doesn’t look quite right then it can be difficult to determine exactly why, and if a composite appears to look good then it can be difficult to know how to make it better.
From the perspective of a professional career, perhaps the most critical thing to understand is that Producers and Directors are not compositors, and they probably won’t have a very strong technical understanding of the process. A Director isn’t likely to look at your scene and give you some technically specific tweak to do. That’s your job (some Directors will but whether or not they make sense is another matter). Although all projects are different, it’s generally up to the individual compositor to decide how little or how much to do to a scene. Producers and Directors might point out obvious things such as adding shadows and reflections, but beyond that it’s up to you.
What Producers and Directors will do, however, is look at your work and decide if it’s any good. If a composite looks average then they don’t need the technical knowledge to know why, they’ll just blame you. You’ve probably heard the saying ‘I don’t know much about art, but I know what I like”… in post-production it’s “I don’t know much about After Effects, but I know what looks good”.
With this in mind, the myth-of-the-single-click is exposed as a true myth – something totally unachievable- because compositing a finished scene together involves so much more than just keying.
Over the years I have developed a mental checklist of things I consider when I’m doing greenscreen compositing, and without writing a book, here are some of them:
Colour Matching
Matching the colour balance of different elements in a composite will get you 90% of the way there. I usually start with the levels plug-in and go through the shadows, mids and then highlights and match everything up, but it’s important to match the overall contrast and saturation of elements too. I prefer to use the ‘tint’ plugin to adjust saturation, rather than the Hue/Saturation plugin. The ‘colour balance’ plugin is also very useful when matching disparate elements, especially with the ‘preserve luminosity’ option turned on. There are other articles and tutorials on the internet that look at colour matching in detail.
Exposure matching
Although you can adjust the different elements in your composite so they have the same colour balance, they still might differ in their brightness and contrast. The exposure dial in the composition window is a quick way of previewing your composite at a range of exposures, and can roughly indicate if a layer is brighter or darker than it should be. The exposure plugin is a quick way to adjust an individual layer without affecting the colour balance. Rendered 3D elements are more likely to have a large tonal range than shot footage, ie. with 3D renders you can reasonably expect black to be 0 and white to be 255, but with real-life footage the darkest areas of the shot may be much higher, or the brightest parts lower.
Stabilising movement
In the olden days, when people shot on film, the image would have a slight camera weave depending on how good the equipment was. 16mm footage could move around quite a bit, and so if were compositing various 16mm elements together you really had to stabilise them all beforehand. But even footage shot on HD or RED might have little movements in it if someone bumps the camera, or the cameraman breathes heavily, and any little bumps or kinks should be smoothed out.
Grain matching
If your footage originated on film then it will have film grain, but even digital cameras have a signature ‘digital noise’ and compression artefacts. If you want elements in your composite to match then you’ll need to compensate for these differences – removing noise from noisy sources and adding noise to clean sources. I generally use the After Effects “remove grain” plug-in with temporal sampling turned on. If you have 3D rendered elements to work with then they’ll be really clean and you’ll need to add grain or noise to them.
Foreground/Background masking
Parts of the background plate might have to appear in front of the foreground plate. This will usually involve rotoscoping the background element out. I find this commonly happens with tree branches or foliage that’s blowing around.
Blur / Sharpness
Footage from different sources will probably have differing amounts of sharpness. 3D renders will be razor sharp, de-interlaced video will be really soft. Film will be softer than video. If you are integrating 3D renders with video then you’ll probably have to blur it more than you expect.
Motion blur
The amount of motion blur in different layers can be different, especially is you have 3D rendered elements that haven’t been rendered with motion blur. Reel Smart Motion Blur is an invaluable tool here.
Depth of Field
If your background plate has a shallow depth of field, then any elements added to the scene will need to match the blurriness of the focal plane they’re in. If there’s a pull-focus of the background plate during the shot then all the elements in your composite will need to adjust accordingly. The ‘box blur’ effect is a fast way of making an element look out of focus. For more complex scenes, you may need to get creative and create a depth map for use with the AE Lens Blur effect, but ideally you’ll have the Frischluft Lenscare plugins which look gorgeous and are faster than the AE stock equivalent. If you’ve added shadows to the scene, perhaps by duplicating a keyed foreground element and filling it with black, the shadow may need to be more blurry at the top than at the bottom. Combining a ramp and the compound blur can achieve this.
Shadows & Reflections
Things cast shadows. So add them if you need to, and the same goes for reflections. You can make shadows look more realistic by precomping them and using the background plate as a displacement map, and reflections can be subtly distorted by scaling the X&Y independently, skewing with the ‘transform’ effect, or even using the liquify effect, to match the surface they’re on. Reflections in water can be distorted by using the water as a displacement map for the reflection layer.
Light wrap
As well as being a plugin that’s part of the Composite Wizard and Key Correct Pro collections, there are various tutorials on the internet about how to add light-wrap without 3rd party plugins. Basically, light areas of the background plate should wrap around the edges of your foreground element.
Interlayer blur
This is a technique that is described in more detail in the ‘Composite Wizard’ manual, but it’s a pain to set up properly. The idea is that the edge/boundary between your foreground/background layers will be blurred together. A quicker method than the CW one is to use an edge-detection plugin to create an outline of your foreground layer, and use it as a matte for an adjustment layer with a blur effect. Very subtle…
Light source matching
If different elements have been lit in different ways, from different directions and with different lights then it’s really hard to adjust them to match. The same goes for 3D renders that may have been lit in a fairly simple way, and have to be integrated into a real-life background. Unfortunately you can’t re-light real-life footage in post perfectly, but you can crudely darken areas of a layer by adding masks, using the fill effect, and feathering the edges and adjusting the fill opacity. To do it more effectively you will have to precompose the layer and rotoscope coloured solids with different transfer modes onto your footage. It’s not going to be perfect, but it will improve the overall composite.
Lens Flares
I love lens flares. If your background plate has light sources or hotspots, you can track them and apply the tracking data to lens flares which are rendered over your foreground layer. Even if I am working with a purely CG scene, I will use the 3D camera data with the toComp expression and have multiple lens flares in the scene. As the camera moves, and the lens flares react accordingly, they add a genuine sense of depth to the scene. I’m not talking about huge, gratuitous 100% opacity flares, but soft, blurry, subtle layers that add some moving light to the scene.
Fog effects
Smoke, or even fractal noise, can not only add character to a scene but it can help tie disparate elements together. By using expressions and the collapse-transformation switch, 3D fractal noise can exist within the scene, in true 3D space, and animate convincingly although if you turn it up too much it looks like you’re remaking Jekyll & Hyde, in 19th Century London.
Global Adjustments
Adjustment layers at the top of a composite can be used to add noise or grain back to the entire scene. You can also add camera movement back into a composited scene, and subtle zooms or pans can really help ‘sell’ the composite.
And that’s just the stuff I can think of.
In part 2 I promised a look at the most challenging key I’ve ever faced, and so to keep my promise I’ve dug out a still-frame:
Oddly enough, Keylight managed a half-decent result and a lot of choking helped to produce a useable result with minimum roto.
Looking back to part 1, there’s no doubt that having great footage makes things easier. But the point of part 2 was that it’s naive to expect keying to be simple, quick and easy. And that’s because compositing is an involved and complex art form, and the list of things that can make up a successful composite is not only limited to the things I’ve listed above.
So the myth-of-the-single-click key is exactly that – a myth.